Getting Started Guide - Clustered Cache in Java SE

Clustering Infinispan is simple. Under the covers, Infinispan uses JGroups as a network transport, and JGroups handles all the hard work of forming a cluster.

Sharing JGroups channels

By default all caches created from a single CacheManager share the same JGroups channel and multiplex RPC messages over it. In this example caches 1, 2 and 3 all use the same JGroups channel:

EmbeddedCacheManager cm = // get CacheManager from somewhere
Cache<Object, Object> cache1 = cm.getCache("replSyncCache");
Cache<Object, Object> cache2 = cm.getCache("replAsyncCache");
Cache<Object, Object> cache3 = cm.getCache("invalidationSyncCache");

clustered-cache quickstart

All the code discussed in this tutorial is available in the clustered-cache quickstart.

Running Infinispan in a cluster

It is easy set up a clustered cache. This tutorial will show you how to create two nodes in different processes on the same local machine. The quickstart follows the same structure as the embedded-cache quickstart, using Maven to compile the project, and a main method to launch the node.

If you are following along with the quickstarts, you can try the examples out. First, compile the project:

$> mvn clean compile dependency:copy-dependencies -DstripVersion

The quickstart contains two examples of a clustered cache, one in replication mode and one distribution mode. To run the replication mode example, we need to launch both nodes from different consoles. For the first node:

$> java -cp target/classes/:target/dependency/* org.infinispan.quickstart.clusteredcache.replication.Node0

And for the second node:

$> java -cp target/classes/:target/dependency/* org.infinispan.quickstart.clusteredcache.replication.Node1

You should see JGroups and Infinispan start up on both consoles, and after about 15s the cache entry log message appear on the console of the first node.

To run the distribution mode example, we need to launch three nodes from different consoles. For the first node:

$> java -cp target/classes/:target/dependency/* org.infinispan.quickstart.clusteredcache.distribution.Node0

And for the second node:

$> java -cp target/classes/:target/dependency/* org.infinispan.quickstart.clusteredcache.distribution.Node1

And for the third node:

$> java -cp target/classes/:target/dependency/* org.infinispan.quickstart.clusteredcache.distribution.Node2

You should see JGroups and Infinispan start up on both consoles, and after about 15s see the 10 entries added by third node distributed to the first and second nodes.

clustered-cache quickstart architecture

Logging changes to the cache

An easy way to see what is going on with your cache is to log mutated entries. An Infinispan listener is notified of any mutations:

LoggingListener.java

18.  * You should have received a copy of the GNU Lesser General Public
19.  * License along with this software; if not, write to the Free
20.  * Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA
21.  * 02110-1301 USA, or see the FSF site: http://www.fsf.org.
22.  */
23. package org.infinispan.quickstart.clusteredcache.util;
24. 
25. import org.infinispan.notifications.Listener;
26. import org.infinispan.notifications.cachelistener.annotation.CacheEntryCreated;
27. import org.infinispan.notifications.cachelistener.annotation.CacheEntryModified;
28. import org.infinispan.notifications.cachelistener.annotation.CacheEntryRemoved;
29. import org.infinispan.notifications.cachelistener.annotation.TopologyChanged;
30. import org.infinispan.notifications.cachelistener.event.CacheEntryCreatedEvent;
31. import org.infinispan.notifications.cachelistener.event.CacheEntryModifiedEvent;
32. import org.infinispan.notifications.cachelistener.event.CacheEntryRemovedEvent;
33. import org.infinispan.notifications.cachelistener.event.TopologyChangedEvent;

Listeners methods are declared using annotations, and receive a payload which contains metadata about the notification. Listeners are notified of any changes. Here, the listeners simply log any entries added or removed.

The replication mode example contains two nodes, each of which are started in a separate process. The nodes are very simple, Node0 starts up, registers a listener that logs any changes, and waits for the cluster to form. Node1 starts up, waits for the cluster to form, and then adds an entry. The interesting work happens in the common super class, examined in Configuring a replicated data-grid.

Waiting for the cluster to form

Infinispan only replicates data to nodes which are already in the cluster. If a node is added to the cluster after an entry is added, it won't be replicated there. In order to see replication take effect, we need to wait until Both nodes make use of the utility class ClusterValidation, calling it's waitForClusterToForm to achieve this. We won't dig into how this works here, but if you are interested take a look at the code.

The distribution mode example contains three nodes, each of which are started in a separate process. The nodes are very simple, Node0 and Node1 start up, register listeners that logs any changes, and wait for the cluster to form. Node2 starts up, waits for the cluster to form, and then adds 20 entries. Each entry get's distributed to it's owners, and you will see some entries add on Node0 and some on Node1. You'll notice that Node2 gets notified of all adds - this is just because it is the node which adds the entry, it doesn't reflect that the fact that all these entries are stored there! The interesting work happens in the common super class, examined in Configuring a distributed data-grid.

Configuring the cluster

First, we need to ensure that Infinispan is cluster aware. Infinispan provides a default configuration for a clustered cache:Code Snippet error: Unable to retrieve the URL: https://github.com/infinispan/infinispan-quickstart/raw/master/clustered-cache/src/main/java/org/infinispan/quickstart/clusteredcache/replication/AbstractNode.java status code: 404.

GlobalConfiguration.getClusteredDefault() is a quick way to get a preconfigured, cluster-aware GlobalConfiguration and can be used as a starting point to fine tuning the configuration.

Tweaking the cluster configuration for your network

Depending on your network setup, you may need to tweak your JGroups set up. JGroups is configured via an XML file; the file to use can be specified via the GlobalConfiguration:Code Snippet error: Unable to retrieve the URL: https://github.com/infinispan/infinispan-quickstart/raw/master/clustered-cache/src/main/java/org/infinispan/quickstart/clusteredcache/replication/AbstractNode.java status code: 404.

The JGroups documentation provides extensive advice on getting JGroups working on your network. If you are new to configuring JGroups, you may get a little lost, so you might want to try tweaking these configuration parameters:

Using the system property -Djgroups.bind_addr="127.0.0.1" causes JGroups to bind only to your loopback interface, meaning any firewall you may have configured won't get in the way. Very useful for testing a cluster where all nodes are on one machine.

TODO - add more tips!

You can also configure the JGroups properties to use in Infinispan's XML configuration:Code Snippet error: Unable to retrieve the URL: https://github.com/infinispan/infinispan-quickstart/raw/master/clustered-cache/src/main/resources/infinispan-replication.xml status code: 404.

Configuring a replicated data-grid

In replicated mode, Infinispan will store every entry on every node in the grid. This offers high durability and availability of data, but means the storage capacity is limited by the available heap space on the node with least memory.

The cache should be configured to work in replication mode (either synchronous or asynchronous), and can otherwise be configured as normal. For example, if you want to configure the cache programatically:Code Snippet error: Unable to retrieve the URL: https://github.com/infinispan/infinispan-quickstart/raw/master/clustered-cache/src/main/java/org/infinispan/quickstart/clusteredcache/replication/AbstractNode.java status code: 404.

You can configure an identical cache using XML:

cfg.xml:Code Snippet error: Unable to retrieve the URL: https://github.com/infinispan/infinispan-quickstart/raw/master/clustered-cache/src/main/resources/infinispan-replication.xml status code: 404.Code Snippet error: Unable to retrieve the URL: https://github.com/infinispan/infinispan-quickstart/raw/master/clustered-cache/src/main/java/org/infinispan/quickstart/clusteredcache/replication/AbstractNode.java status code: 404.

Configuring a distributed data-grid

In distributed mode, Infinispan will store every entry on a subset of the nodes in the grid (controlled by the parameter numOwners, which controls how many owners each entry will have). Compared to replication, distribution offers increased storage capacity, but with reduced availability (increased latency to access data) and durability. Adjusting the number of owners allows you to obtain the trade off between space, durability and availability.

Infinispan also offers a topology aware consistent hash which will ensure that the owners of entries are located in different data centers, racks and nodes to offer improved durability in case of node or network outages.

The cache should be configured to work in distibuted mode (either synchronous or asynchronous), and can otherwise be configured as normal. For example, if you want to configure the cache programatically:Code Snippet error: Unable to retrieve the URL: https://github.com/infinispan/infinispan-quickstart/raw/master/clustered-cache/src/main/java/org/infinispan/quickstart/clusteredcache/distribution/AbstractNode.java status code: 404.